Video display system

ABSTRACT

A video display system that includes a display device and an image processing device. The display device is mounted on the head of a user for use. The image processing device generates a plurality of unsynthesized images having different resolutions that are to be used for display by the display device, and transmits the generated unsynthesized images to the display device. The display device receives the unsynthesized images transmitted from the image processing device, and displays a display image that is obtained by synthesizing the unsynthesized images.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation application of U.S. patent application Ser. No.16/060,999, accorded a filing date of Jun. 11, 2018, which is a nationalphase application of International Application No. PCT/JP2017/004065,filed Feb. 3, 2017, which claims priority to Japanese Application No. JP2016-022622, filed Feb. 9, 2016, the entire disclosures of which arehereby incorporated by reference.

TECHNICAL FIELD

The present invention relates to a video display system that displaysvideo on a display device mounted on the head of a user for use, animage processing device and a display device included in the videodisplay system, an image processing method, and a program.

BACKGROUND ART

A known technology called “Foveated Rendering” renders a region ofinterest (a region estimated to be gazed by the user) within an image ata higher resolution than the outside of the region of interest. Thistechnology makes it possible to generate an image by rendering theregion of interest at a high resolution with a lower processing loadthan when the image is entirely rendered at the high resolution.

SUMMARY Technical Problems

When the above technology is used, the image eventually displayed on adisplay device needs to be entirely at a high resolution in accordancewith the resolution of the region of interest. Further, it is desirablethat a display device mounted on the head of the user for use, such as ahead-mounted display, display video at a relatively high frame rate. Inorder to display video at a high resolution and at a high frame rate asmentioned above, an image processing device for generating a displayimage needs to transmit image data to the display device at a high datatransmission rate.

The present invention has been made in view of the above circumstances.An object of the present invention is to provide a video display system,an image processing device, a display device, an image processingmethod, and a program that are capable of relatively reducing a datatransmission amount when the display device displays an image indicatinga high-resolution region of interest.

Solution to Problems

A video display system according to the present invention includes adisplay device mounted on the head of a user for use and an imageprocessing device for supplying video to be displayed on the displaydevice. The image processing device includes an image generation sectionand a transmission section. The image generation section generates aplurality of unsynthesized images having different resolutions that areto be used for display by the display device. The transmission sectiontransmits each of the unsynthesized images to the display device. Thedisplay device includes a reception section and a display controlsection. The reception section receives the unsynthesized imagestransmitted from the image processing device. The display controlsection displays a display image that is obtained by synthesizing theunsynthesized images.

An image processing device according to the present invention suppliesvideo to a display device mounted on the head of a user for use. Theimage processing device includes an image generation section and atransmission section. The image generation section generates a pluralityof unsynthesized images having different resolutions that are to besynthesized to obtain a display image to be displayed on the displaydevice. The transmission section transmits each of the unsynthesizedimages to the display device.

A display device according to the present invention is mounted on thehead of a user for use and connected to an image processing device thatsupplies video to the display device. The display device includes areception section and a display control section. The reception sectionreceives a plurality of unsynthesized images that are transmitted fromthe image processing device and different in resolution. The displaycontrol section displays a display image that is obtained bysynthesizing the unsynthesized images.

An image processing method according to the present invention is usedfor supplying video to a display device mounted on the head of a userfor use. The image processing method includes: a step of generating aplurality of unsynthesized images having different resolutions that areto be synthesized to obtain a display image to be displayed on thedisplay device; and a step of transmitting each of the unsynthesizedimages to the display device.

A program according to the present invention controls an imageprocessing device that supplies video to a display device mounted on thehead of a user for use. The program causes the image processing deviceto function as an image generation section and as a transmissionsection. The image generation section generates a plurality ofunsynthesized images having different resolutions that are to besynthesized to obtain a display image to be displayed on the displaydevice. The transmission section transmits each of the unsynthesizedimages to the display device. The program may be provided by beingstored in a computer-readable, non-transitory information storagemedium.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an overall configuration of avideo display system according to an embodiment of the presentinvention.

FIG. 2 is a functional block diagram illustrating functions implementedby the video display system according to the embodiment of the presentinvention.

FIG. 3 is a diagram illustrating an example of an overall image P0.

FIG. 4 is a diagram illustrating an example of a first partial image P1.

FIG. 5 is a diagram illustrating an example of a second partial imageP2.

FIG. 6 is a diagram illustrating the contents of data that istransmitted from an image processing device to a display device.

FIG. 7 is a diagram illustrating an example of a frame image to bedisplayed on the display device.

FIG. 8 is a sequence diagram illustrating an example of a flow of aprocess performed by the video display system.

FIG. 9 is a diagram illustrating an exemplary positional relationshipbetween unsynthesized images within a frame image.

DESCRIPTION OF EMBODIMENT

An embodiment of the present invention will now be described withreference to the accompanying drawings.

FIG. 1 is a block diagram illustrating a configuration of a videodisplay system 1 according to an embodiment of the present invention. Asillustrated in FIG. 1, the video display system 1 includes an imageprocessing device 10, an operating device 20, a relay device 30, and adisplay device 40.

The image processing device 10 generates and supplies an image that isto be displayed by the display device 40. The image processing device 10may be, for example, a home-use game machine, a portable game machine, apersonal computer, a smartphone, or a tablet. As illustrated in FIG. 1,the image processing device 10 includes a control section 11, a storagesection 12, and an interface section 13.

The control section 11 includes at least one processor, such as acentral processing unit (CPU), and executes a program stored in thestorage section 12 to perform various information processes. Concreteexamples of the processes performed by the control section 11 in thepresent embodiment will be described later. The storage section 12includes at least one memory device, such as a random access memory(RAM), and stores the program to be executed by the control section 11and data to be processed by the program.

The interface section 13 is an interface for data communication betweenthe operating device 20 and the relay device 30. The image processingdevice 10 is wire-connected or wirelessly connected to the operatingdevice 20 and the relay device 30 through the interface section 13. Morespecifically, the interface section 13 may include a high-definitionmultimedia interface (HDMI) (registered trademark) or other multimediainterface in order to transmit video and audio supplied from the imageprocessing device 10 to the relay device 30. Further, the interfacesection 13 includes a Bluetooth (registered trademark), universal serialbus (USB), or other data communication interface. Through the datacommunication interface, the image processing device 10 receives variousinformation from the display device 40 via the relay device 30 andtransmits, for example, a control signal. Further, the image processingdevice 10 receives an operation signal transmitted from the operatingdevice 20 through the data communication interface.

The operating device 20 is, for example, a controller or keyboard for ahome-use game machine, and receives an operation input from a user. Theoperating device 20 transmits to the image processing device 10 a signalindicative of the contents of the operation input received from theuser.

The relay device 30 is connected to the display device 40. The relaydevice 30 receives image data supplied from the image processing device10, and transmits the received image data to the display device 40. Inthis instance, the relay device 30 may perform, for example, acorrection process on the supplied image data as needed to canceldistortion caused by an optical system of the display device 40, andoutput the corrected image data. Further, in addition to the image data,the relay device 30 relays audio data, control signals, and variousother information exchanged between the image processing device 10 andthe display device 40. The present embodiment assumes that the relaydevice 30 wirelessly transmits and receives data to and from the displaydevice 40.

The display device 40 displays video based on the image data receivedfrom the relay device 30 in order to permit the user to view thedisplayed video. The present embodiment assumes that the display device40 is mounted on the head of the user for use and adapted to permit botheyes to view the video. That is to say, the display device 40 forms avideo image in front of each of the right and left eyes of the user. Thedisplay device 40 may be configured so as to be able to display athree-dimensional video image based on binocular parallax. Asillustrated in FIG. 1, the display device 40 includes a processing unit41, a communication interface 42, a buffer memory 43, a video displayelement 44, an optical element 45, and a rear camera 46.

The processing unit 41 includes, for example, an integrated circuit,generates a frame image (display image) on the basis of the image datathat is received from the image processing device 10 through the relaydevice 30, and supplies the generated frame image to the video displayelement 44. By repeating this process at a predetermined frame rate, theprocessing unit 41 causes the video display element 44 to display video.

The communication interface 42 is used to establish data communicationwith the relay device 30. The communication interface 42 includes, forexample, an antenna and communication circuit for wirelesscommunication. The image data received from the relay device 30 by thecommunication interface 42 is temporarily stored in the buffer memory43. The processing unit 41 generates a frame image on the basis of theimage data stored in the buffer memory 43.

The video display element 44 is, for example, an organicelectroluminescence (EL) display panel or a liquid-crystal displaypanel, and displays video based on a video signal supplied from theprocessing unit 41. The video display element 44 displays two videoimages, namely, a left-eye video image and a right-eye video image. Thevideo display element 44 may be a single display element for displayingthe left- and right-eye video images side by side or two displayelements for displaying video images on an independent basis. Further,for example, a well-known smartphone may be used as the video displayelement 44. Furthermore, the display device 40 may be a retinalirradiation (retinal projection) display device that projects a videoimage directly to the retinas of the user. In this case, the videodisplay element 44 may include, for example, a laser that emits lightand a micro electro mechanical systems (MEMS) mirror that scans theemitted light.

The optical element 45 is, for example, a holographic optical element, aprism, or a half mirror. Disposed in front of the eyes of the user, theoptical element 45 transmits or refracts video light emitted from thevideo display element 44 and causes the video light to become incidenton the left and right eyes of the user. More specifically, the left-eyevideo image displayed by the video display element 44 becomes incidenton the left eye of the user through the optical element 45, and theright-eye video image becomes incident on the right eye of the userthrough the optical element 45. With the display device 40 mounted onthe user's head, the user is then able to view the left-eye video imagewith the left eye and view the right-eye video image with the right eye.

The rear camera 46 is disposed so as to be able to capture an image of arear view from the display device 40 (that is, capture an image of theuser), and is used to capture an image of each of the left and righteyes of the user. The image captured by the rear camera 46 istransmitted to the image processing device 10 through the relay device30.

Functions implemented by the video display system 1 will now bedescribed with reference to FIG. 2. As illustrated in FIG. 2, the imageprocessing device 10 functionally includes a point-of-interestidentification section 51, an image generation section 52, and an imagedata transmission section 53. These functions are implemented when thecontrol section 11 executes a program stored in the storage section 12.The program may be provided to the image processing device 10 through acommunication network such as the Internet or provided by being storedin a computer-readable information storage medium such as an opticaldisk. Meanwhile, the processing unit 41 in the display device 40functionally includes a timing control section 54, an image dataselection section 55, and a display control section 56. Some or all ofthese functions may be implemented by software or implemented byhardware such as an electronic circuit.

The point-of-interest identification section 51 identifies a position (apoint of interest) within a display region that is gazed by a user usingthe display device 40. More specifically, the point-of-interestidentification section 51 acquires an image captured by the rear camera46 from the display device 40, and analyzes the acquired captured imageto identify the gaze direction of the user. The point of interest is aposition within the display region that corresponds to the gazedirection.

The image generation section 52 generates an image that is to bedisplayed on the display device 40. As a concrete example, it is assumedhere that the image drawn by the image generation section 52 depicts athree-dimensional virtual space where various objects are disposed.However, the image generation section 52 is not limited to theabove-described type, and may draw various other images. Further, it isassumed for the sake of simplicity that the display device 40 displaysthe same frame image as a left-eye image and a right-eye image. However,the video display system 1 according to the present embodiment maydisplay different frame images as the left- and right-eye images toachieve three-dimensional display based on parallax. In such a case, theimage processing device 10 and the display device 40 should parallellyperform the following described process for each of left and right frameimages.

In the present embodiment, the image generation section 52 generates aplurality of images having different resolutions for one frame image tobe displayed on the display device 40. These images draw the same targetat different resolutions in such a manner that the higher theresolution, the smaller the drawn region. As described later, theseimages are synthesized into one frame image in the display device 40 anddisplayed by the video display element 44. The images generated by theimage generation section 52 and eventually synthesized into one frameimage are hereinafter referred to as the unsynthesized images. Further,as a concrete example, it is hereinafter assumed that the imagegeneration section 52 generates three different unsynthesized images. Inorder from the lowest resolution to the highest, the three differentunsynthesized images are an overall image P0, a first partial image P1,and a second partial image P2.

FIGS. 3, 4 and 5 are diagrams illustrating examples of the threedifferent unsynthesized images. As illustrated in FIG. 3, the overallimage P0 is displayed on the assumption that it is displayed in thewhole display region of the display device 40 at a lower resolution thanthe first partial image P1 and the second partial image P2. Meanwhile,as illustrated in FIG. 4, the first partial image P1 is obtained bydrawing a partial region within the overall image P0 at a higherresolution than the overall image P0. The second partial image P2, whichis illustrated in FIG. 5, is obtained by drawing a partial regionsmaller than the first partial image P1 at an even higher resolutionthan the first partial image P1. It is assumed that the second partialimage P2 is displayed at the same resolution as a frame image actuallydisplayed on the display device 40.

Regions to be drawn as the first partial image P1 and the second partialimage P2 are determined on the basis of the point of interest identifiedby the point-of-interest identification section 51. More specifically,the partial images are obtained by drawing regions centered around thepoint of interest within the overall image P0. If, for example, the userwearing the display device 40 gazes a point that is upper right of thecenter of the display region, the first partial image P1 and the secondpartial image P2 are obtained by drawing an upper right region withinthe overall image P0. FIGS. 4 and 5 illustrate a case where the point ofinterest is at the center of the overall image P0, and each of thepartial images corresponds to the central region of the overall imageP0.

As described above, the unsynthesized images drawn by the imagegeneration section 52 are such that the larger the target region, thelower the resolution. Therefore, the total amount of data in the overallimage P0, the first partial image P1, and the second partial image P2 issmaller than the amount of data in an eventually displayed frame image(i.e., an image obtained by drawing the whole region covered by theoverall image P0 at the same resolution as the second partial image P2).

The image data transmission section 53 transmits unsynthesized imagedata generated by the image generation section 52 to the relay device30. In this instance, the image data transmission section 53individually transmits the overall image P0, the first partial image P1,and the second partial image P2 while they are unsynthesized. Thisensures that the amount of data to be transmitted from the imageprocessing device 10 to the display device 40 in order to display oneframe image is smaller than when a high-resolution, synthesized frameimage is transmitted.

In the present embodiment, instead of the image processing device 10,which generates the unsynthesized images, the display device 40synthesizes the unsynthesized images to generate a frame image to beactually displayed. Therefore, it is assumed that the image datatransmission section 53 transmits additional information together withthe unsynthesized image data. The additional information includesvarious items of information indicating how the display device 40 shouldsynthesize the unsynthesized images.

More specifically, before transmitting the unsynthesized images, theimage data transmission section 53 attaches to each of the unsynthesizedimages the additional information indicative of the position of anunsynthesized image within a frame image, the size of the unsynthesizedimage within the frame image, the ratio of magnification, and otherparameters necessary for processing the unsynthesized image. FIG. 6 is aschematic diagram illustrating the contents of data transmitted from theimage data transmission section 53. FIG. 6 illustrates theabove-described additional information attached to the unsynthesizedimages. Leading coordinates in FIG. 6 are examples of additionalinformation indicative of the position of an unsynthesized image withina frame image. The leading coordinates represent position coordinates ofan upper left pixel of an unsynthesized image within a frame image.

Image data transmitted from the image data transmission section 53 isinputted to the display device 40 through the relay device 30. Thecommunication interface 42 of the display device 40 receivesunsynthesized image data from the relay device 30 and temporarily storesthe received unsynthesized image data in the buffer memory 43. Thebuffer memory 43 stores the unsynthesized image data received by thecommunication interface 42 on an as-is basis. Therefore, the buffermemory 43 does not need to have an adequate capacity for storing thewhole of one frame image displayed by the video display element 44.

The timing control section 54 supplies a synchronization signal forcontrolling the timing of frame image display to the image dataselection section 55 and to the display control section 56. The imagedata selection section 55 and the display control section 56 operatewith timing derived from the synchronization signal supplied from thetiming control section 54, and thus perform processing in synchronismwith each other.

The image data selection section 55 and the display control section 56cooperate to perform a process of synthesizing the unsynthesized imagesreceived from the image processing device 10 and generating a frameimage to be displayed. More specifically, the image data selectionsection 55 determines an unsynthesized image to be used to determine thepixel value of each pixel in the frame image. Basically, the image dataselection section 55 selects the highest-resolution unsynthesized imagefrom among the unsynthesized images covering a pixel position ofinterest, and uses the selected highest-resolution unsynthesized image.For example, for pixels within a region corresponding to the secondpartial image P2 at the center of the frame image, the second partialimage P2 is selected and used. Conversely, for a peripheral region ofthe frame image that is not covered by the first partial image P1 andthe second partial image P2, the overall image P0 is selected and used.The image data selection section 55 makes the above-described selectionsby referencing the position and size information included in theadditional information that is transmitted together with theunsynthesized images by the image data transmission section 53. Then,for each pixel in the frame image, the image data selection section 55reads from the buffer memory 43 the pixel value information on acorresponding pixel in the selected unsynthesized image, and outputs theread pixel value information to the display control section 56.

On the basis of image data on the unsynthesized image selected by theimage data selection section 55, the display control section 56determines the pixel value of each pixel in the frame image. Then, thedisplay control section 56 supplies to the video display element 44 avideo signal for displaying the frame image including determined pixelvalues. Consequently, the video display element 44 is able to displaythe frame image that is obtained by synthesizing the unsynthesizedimages. Before synthesizing the unsynthesized images, the displaycontrol section 56 magnifies low-resolution unsynthesized images. Themagnification ratio for such magnification is determined by amagnification ratio parameter in the additional information illustrated,for example, in FIG. 6.

When synthesizing the different unsynthesized images, the displaycontrol section 56 may perform image processing, for example, in such amanner as to blur and make indistinct the boundaries between thedifferent unsynthesized images. When such image processing is to beperformed, the details of image processing may be determined by usingparameters included in the additional information transmitted from theimage data transmission section 53.

FIG. 7 illustrates an example of a frame image that is obtained bysynthesizing three different unsynthesized images exemplified in FIGS. 3to 5. This frame image has the same resolution as the second partialimage P2 as a whole; however, a peripheral portion of the image isobtained by magnifying the low-resolution overall image P0. Meanwhile, aregion estimated to be gazed by the user is generated on the basis ofthe high-resolution second partial image P2. Therefore, the user feelsthat the frame image is displayed at a high resolution.

A flow of a process performed by the video display system 1 to displayone frame image will now be described with reference to the flowchart ofFIG. 8.

First of all, the display device 40 transmits an image captured by therear camera 46 to the image processing device 10 (step S1). Thepoint-of-interest identification section 51 in the image processingdevice 10 then identifies the point of interest of the user by using thecaptured image transmitted in step S1 (step S2).

Next, the image generation section 52 generates three differentunsynthesized images by using the information on the point of interestidentified in step S2 (step S3). Subsequently, the image datatransmission section 53 adds the additional information to the generatedthree different unsynthesized images and transmits the resultingcombination of images and additional information to the display device40 (step S4).

The communication interface 42 of the display device 40 stores in thebuffer memory 43 image data on the unsynthesized images transmitted instep S4 (step S5). The processing unit 41 then generates a frame imageby synthesizing the three different unsynthesized images stored in thebuffer memory 43 in step S5 (step S6), and causes the video displayelement 44 to display the generated frame image (step S7).

The video display system 1 displays video by repeatedly performing theabove-described process to update the frame image. The flowchart assumesthat each time the frame image is updated, the image captured by therear camera 46 is transmitted from the display device 40 to the imageprocessing device 10 to update the point of interest. However, thepoint-of-interest identification section 51 may alternatively update thepoint of interest at longer time intervals. In such an alternative case,the image captured by the rear camera 46 need not be transmitted eachtime the frame image is updated. Thus, steps S1 and S2 are performedless frequently.

As described above, the video display system 1 according to the presentembodiment is configured so that the image processing device 10transmits a plurality of different unsynthesized images having differentresolutions to let the display device 40 synthesize the unsynthesizedimages to obtain and display a frame image. Consequently, the amount ofdata to be transmitted to display one frame image can be reduced ascompared with a case where a synthesized frame image is transmitted fromthe image processing device 10 to the display device 40. As a result,the display device 40 is able to display video indicating ahigh-resolution region of interest at a relatively high frame rate evenif a great communication bandwidth is not provided between the imageprocessing device 10 and the display device 40.

The foregoing description assumes that the first partial image P1 andthe second partial image P2 are rectangular images having the same shapeas the overall image P0 but different sizes from the overall image P0.However, the partial images are not limited to such a shape, and mayhave a square shape, a circular shape, or other shape different fromthat of the whole display region. Further, the image processing device10 may select a partial image shape from among a plurality ofcandidates. In such an instance, the additional information includesinformation that specifies the partial image shape. The display device40 references such additional information to identify what region of theframe image corresponds to a received partial image. Meanwhile, theoverall image P0, too, is not limited to a rectangular shape, and mayhave a circular shape or other shape based on the displaycharacteristics of the display device 40.

The foregoing description assumes that the positions of partial imageswithin a frame image are determined on the basis of the point ofinterest, which is gazed by the user. Therefore, when the gaze directionof the user changes, the positions of the partial images within theoverall image also change accordingly. However, the present embodimentis not limited to such a configuration. The point of interest mayalternatively be a fixed point such as the center of the display region.In such a case, the point-of-interest identification section 51 is notrequired.

The foregoing description assumes that the unsynthesized images to besynthesized to obtain one frame image include two different partialimages. Alternatively, however, one partial image or three or morepartial images may be included. In any case, the amount of data to betransmitted can be reduced by determining the resolution and size ofeach partial image in such a manner that the higher the resolution of apartial image, the smaller the corresponding region within a frameimage.

The foregoing description assumes that a synthesis process is performedafter three unsynthesized images are completely received. Alternatively,however, the processing unit 41 may perform the synthesis process on areceived portion of the unsynthesized images while the communicationinterface 42 is receiving the unsynthesized images. In this case, at atime when leading portions of the unsynthesized images are stored in thebuffer memory 43, the processing unit 41 acquires and synthesizes thestored portions, and supplies the resulting portions to the videodisplay element 44 as a video signal. Upon receiving the subsequentportions of the unsynthesized images, the communication interface 42overwrites portions processed by the processing unit 41 with thereceived subsequent portions and stores the resulting portions. Whensuch parallel processing is performed, it is necessary that the buffermemory 43 have only a capacity sufficient for storing certain portionsof the unsynthesized images. That is to say, the buffer memory 43 doesnot require a capacity for storing the whole data on all theunsynthesized images.

When the above-described parallel processing is to be implemented, it ispreferable that the image data transmission section 53 divide eachunsynthesized image into a plurality of transmission units (hereinafterreferred to as the blocks) and transmit image data on an individualblock basis. Here, it is assumed as a concrete example that theprocessing unit 41 gradually generates a frame image, starting with thehighest upper side line and working sequentially towards the lowestlower side line, by regarding each of laterally arranged pixel rows(lines) within a frame image as a unit of processing. If, in thisinstance, the point of interest is at or below the center of the displayregion, the highest line of the frame image is generated on the basis ofthe overall image P0 as illustrated in FIG. 7 so that the data on thefirst partial image P1 and the second partial image P2 are not usedimmediately after the start of frame image generation. Subsequently,when the lines at position coordinates Y2 and beyond are to begenerated, the data on the overall image P0 and the first partial imageP1 are used, and when the lines at position coordinates Y1 and beyondare to be generated, the data on all the unsynthesized images arerequired. Therefore, the image data transmission section 53 divides eachof the unsynthesized images into a plurality of blocks and transmits theindividual blocks in the order required for frame image synthesisprocessing. In this instance, each block may correspond to one or morelines of laterally arranged pixel rows within each synthesized image.The image data transmission section 53 sequentially transmits the blocksof each partial image to the display device 40. More specifically, forexample, the image data transmission section 53 first transmits blocksforming the highest portion of the overall image P0 (the blockscorresponding to position coordinates Y3 to Y2), then transmits blocksof the overall image P0 and first partial image P1, which correspond toposition coordinates Y2 to Y1, and subsequently transmits blocks of eachunsynthesized image corresponding to position coordinates Y1 and beyond.In this manner, the display device 40 is able to start generating anddisplaying a frame image starting with the upper portion while receivingthe data on the partial images.

Further, it is preferable in the above instance that the image datatransmission section 53 change the order of individual blocktransmissions in accordance with a change in the point of interest.When, as a concrete example, the point of interest moves upward from thecenter, the positions of the first partial image P1 and second partialimage P2 relative to the overall image P0 also move upward. FIG. 9illustrates an example of a frame image that is obtained in the abovecase by synthesizing three different unsynthesized images. When thepoint of interest moves upward as described above, the upper blocks ofthe first and second partial images P1 and P2 are required immediatelyafter the start of frame image generation as compared with a time pointbefore such a move. Therefore, when the point of interest moves upward,these blocks are preferentially transmitted. Exercising control asdescribed above makes it possible to generate a frame image withoutdelay.

If a plurality of different unsynthesized images are required togenerate one line within a frame image, a low-resolution unsynthesizedimage may be preferentially transmitted. If, for example, a portionbetween position coordinates Y2 to Y1 in the example of FIG. 7 is to besubjected to synthesis, corresponding blocks of the overall image P0 andfirst partial image P1 are basically required. However, thecorresponding block of the overall image P0 is preferentiallytransmitted, and then the block of the first partial image P1 thatcorresponds to the same position is transmitted. Consequently, when thecorresponding block of the first partial image P1 is not transmitted intime due, for instance, to communication delay, using the overall imageP0 makes it possible to generate and display a frame image even if it islow in resolution.

The foregoing description assumes that the image generation section 52itself draws unsynthesized images depicting a three-dimensional space.Alternatively, however, the image generation section 52 is not limitedto the above-described type, and may generate unsynthesized images onthe basis of a prepared high-resolution image. If, for example, videodata is available that includes a frame image having the same resolutionas the frame image to be displayed on the display device 40, the imagegeneration section 52 generates an image having a considerably loweroverall resolution than the available frame image as the overall imageP0. Further, an image obtained by slightly lowering the resolution byclipping a predetermined region of this frame image is handled as thefirst partial image P1, and an image obtained by clipping a smallerpredetermined region than for the first partial image P1 is handled asthe second partial image P2 having the unchanged resolution. The imagedata transmission section 53 transmits each unsynthesized imagegenerated in the above manner to the display device 40. Consequently,even when high-resolution video is to be reproduced, it is possible toreduce the amount of image data to be transmitted from the imageprocessing device 10 to the display device 40.

The foregoing description assumes that the relay device 30 is wirelesslycommunicatively connected to the display device 40. However, theembodiment of the present invention is not limited to such aconfiguration. Alternatively, the image processing device 10 may beconnected to the display device 40 via various communication links.Further, the relay device 30 is not always required. The imageprocessing device 10 may be directly connected to the display device 40.

REFERENCE SIGNS LIST

-   -   1 Video display system    -   10 Image processing device    -   11 Control section    -   12 Storage section    -   13 Interface section    -   30 Relay device    -   40 Display device    -   41 Processing unit    -   42 Communication interface    -   43 Buffer memory    -   44 Video display element    -   45 Optical element    -   51 Point-of-interest identification section    -   52 Image generation section    -   53 Image data transmission section    -   54 Timing control section    -   55 Image data selection section    -   56 Display control section

The invention claimed is:
 1. An image processing device that suppliesvideo to a display device mounted on the head of a user, the imageprocessing device comprising: an image generation section operating togenerate a plurality of unsynthesized images having differentresolutions, the unsynthesized images being to be synthesized to obtaina display image to be displayed on the display device; and atransmission section operating to transmit each of the unsynthesizedimages to the display device; and a point-of-interest identificationsection operating to identify a point of interest, the point of interestbeing within a display region of the display device and gazed by theuser, wherein: the unsynthesized images include a partial image thatcorresponds to a partial region within the display image, thetransmission section transmits to the display device the partial imageand additional information that identifies the corresponding position ofthe partial image within the display image, a display image is generatedby synthesizing the unsynthesized images in such a manner that thepartial image corresponds to the position identified by the additionalinformation, the image generation section generates the partial image insuch a manner that the partial image corresponds to a position withinthe display image that is determined on the basis of the point ofinterest, and the transmission section divides each of the unsynthesizedimages into a plurality of units of transmission, transmits the imagesby each unit of transmission to the display device, and changes theorder of transmission of the units of transmission in accordance withthe point of interest, the units of transmission forming each of theunsynthesized images.
 2. The image processing device of claim 1, whereinthe image generation section operates to generate the plurality ofunsynthesized images having different size dimensions ranging fromlargest to smallest, and different resolutions ranging from highest tolowest, that are to be used for display by the display device.
 3. Theimage processing device of claim 2, wherein a frame image is displayedby the display device that is obtained by synthesizing the unsynthesizedimages; the frame image includes at least a portion of eachunsynthesized image, and the portions of lower resolution than thehighest resolution are magnified and boundaries between the portions areindistinct; and a total amount of data in the unsynthesized images isless than a total amount of data in an image obtained by drawing thelargest unsynthesized image at the resolution of the smallestunsynthesized image.
 4. An image processing method, comprising:supplying video to a display device mounted on the head of a user;generating a plurality of unsynthesized images having differentresolutions that are to be synthesized to obtain a display image to bedisplayed on the display device, where the unsynthesized images includea partial image that corresponds to a partial region within the displayimage; transmitting each of the unsynthesized images to the displaydevice, including transmitting to the display device the partial imageand additional information that identifies the corresponding position ofthe partial image within the display image; generating a display imageby synthesizing the unsynthesized images in such a manner that thepartial image corresponds to the position identified by the additionalinformation; operating to identify a point of interest, the point ofinterest being within a display region of the display device and gazedby the user; generating the partial image in such a manner that thepartial image corresponds to a position within the display image that isdetermined on the basis of the point of interest; dividing each of theunsynthesized images into a plurality of units of transmission;transmitting the images by each unit of transmission to the displaydevice; and changing the order of transmission of the units oftransmission in accordance with the point of interest, the units oftransmission forming each of the unsynthesized images.
 5. The method ofclaim 4, wherein the generating includes generating the plurality ofunsynthesized images having different size dimensions ranging fromlargest to smallest, and different resolutions ranging from highest tolowest, that are to be used for display by the display device.
 6. Themethod of claim 5, wherein: a frame image is displayed by the displaydevice that is obtained by synthesizing the unsynthesized images; theframe image includes at least a portion of each unsynthesized image, andthe portions of lower resolution than the highest resolution aremagnified and boundaries between the portions are indistinct; and atotal amount of data in the unsynthesized images is less than a totalamount of data in an image obtained by drawing the largest unsynthesizedimage at the resolution of the smallest unsynthesized image.
 7. Anon-transitory, computer readable storage medium containing a computerprogram, which when executed by a computer causes the computer to carryout an image processing method, comprising: supplying video to a displaydevice mounted on the head of a user; generating a plurality ofunsynthesized images having different resolutions that are to besynthesized to obtain a display image to be displayed on the displaydevice, where the unsynthesized images include a partial image thatcorresponds to a partial region within the display image; transmittingeach of the unsynthesized images to the display device, includingtransmitting to the display device the partial image and additionalinformation that identifies the corresponding position of the partialimage within the display image; generating a display image bysynthesizing the unsynthesized images in such a manner that the partialimage corresponds to the position identified by the additionalinformation; operating to identify a point of interest, the point ofinterest being within a display region of the display device and gazedby the user; generating the partial image in such a manner that thepartial image corresponds to a position within the display image that isdetermined on the basis of the point of interest; dividing each of theunsynthesized images into a plurality of units of transmission;transmitting the images by each unit of transmission to the displaydevice; and changing the order of transmission of the units oftransmission in accordance with the point of interest, the units oftransmission forming each of the unsynthesized images.